Estimating Criterion-referenced Standards for Multiple-choice Examinations

Four methods were studied for setting a standard on a written examination containing several clinical discipline subtests. The NBME method yielded the most consistent estimate. The Angoff and Ebel methods yielded slightly less consistent estimates. All estimates were more consistent when computed as the average of discipline standards rather than the judges' personal standards. All but the Essential Content method yielded similar and practical estimates of the standard. The Ebel method was not found to be feasible for use with the examination studied; however, incorporating clusters of equally difficult, relevant items and feedback based on the judges' previous judgments into the NBME method yielded subsequent estimates of the standard that were more consistent than, but not significantly different from, the first estimate. The findings suggest the importance of enabling judges to select the items which will be most meaningful to them in expressing their judgments, the benefit of presenting clusters of equally difficult and relevant items and the utility of providing the judges feedback about their judgments and the opportunity to reconsider them. The study also showed that Rasch calibration and equating procedures provide a feasible methodology for expressing judgments based on different item samples on a common measurement scale. ESTIMATING CRITERION-REFERENCED STANDARDS FOR MULTIPLE-CHOICE EXAMINATIONS Francis P. Hughes, Ph.D. Charles F. Schumacher, Ph.D. Benjamin D. Wright, Ph.D.